kuberntes开发教程系列(3)--了解和编写Custom Resources

字数统计: 2.7k阅读时长: 13 min

 2019/09/08   Share

作为kuberntes生态系统中最核心的扩展之一 , Custom Resources(CR) 扮演这重要角色 , 许多成功的项目都离不开CR的使用 ,比如Istio, Linkerd 2.0, AWS App Mesh等等

Kubernetes cluster version 1.7之后, CR作为主要的 Kubernetes API 资源存储在etcd 中 , 如下图所示，如果请求不是以下任何一个请求，则返回到 apiextensions-apiserver，该服务器为通过 CRDs 定义的资源提供服务

请求如果是aggregation的就使用aggregated API servers处理
如果是原生请求则用Native Kubernetes resources处理
如果是自定义资源 , 则用 apiextensions-apiserver处理

一个典型的crd资源如下

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ats.cnat.programming-kubernetes.info
spec:
  group: cnat.programming-kubernetes.info
  names:
    kind: At
    listKind: AtList
    plural: ats
    singular: at
  scope: Namespaced
  subresources:
    status: {}
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true

Discovery Information

kubectl 使用来自 API 服务器的发现信息来发现新资源

我们用kuberntes的verbosity level来看发生了什么

$ kubectl get ats -v=7
... GET https://XXX.eks.amazonaws.com/apis/cnat.programming-kubernetes.info/
                                      v1alpha1/namespaces/cnat/ats?limit=500
... Request Headers:
... Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
      User-Agent: kubectl/v1.14.0 (darwin/amd64) kubernetes/641856d
... Response Status: 200 OK in 607 milliseconds
NAME         AGE
example-at   43s

起初 , kubectl并不知道自定义资源的存在
kubectl向API server的/apis的所有已存在的API groups发起请求
接下来，kubectl 通过 / apis / group 版本组发现端点向 API 服务器询问所有现有 API 组中的资源
然后，kubectl 将给定类型 ats 转换为三重类型
- Group ( cnat.programming-kubernetes.info)
- Version ( v1alpha1)
- Resource (ats)

在最后一步中，发现端点提供了进行转化所需的所有信息

$ http localhost:8080/apis/
{
  "groups": [{
    "name": "at.cnat.programming-kubernetes.info",
    "preferredVersion": {
      "groupVersion": "cnat.programming-kubernetes.info/v1",
      "version": "v1alpha1“
    },
    "versions": [{
      "groupVersion": "cnat.programming-kubernetes.info/v1alpha1",
      "version": "v1alpha1"
    }]
  }, ...]
}

$ http localhost:8080/apis/cnat.programming-kubernetes.info/v1alpha1
{
  "apiVersion": "v1",
  "groupVersion": "cnat.programming-kubernetes.info/v1alpha1",
  "kind": "APIResourceList",
  "resources": [{
    "kind": "At",
    "name": "ats",
    "namespaced": true,
    "verbs": ["create", "delete", "deletecollection",
      "get", "list", "patch", "update", "watch"
    ]
  }, ...]
}

这些都是通过发现 RESTMapper 实现的

kubectl会在 ~/.kubectl目录下把自己发现的所有资源类型缓存下来 , 每过10分钟失效一次,重新开始开始发现, 所有我们自定义的资源会在10分钟后出现在此缓存中

自定义资源的编写

自定义资源就是CRD 格式如下:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: name
spec:
  group: group name
  version: version name
  names:
    kind: uppercase name
    plural: lowercase plural name
    singular: lowercase singular name # defaulted to be lowercase kind
    shortNames: list of strings as short names # optional
    listKind: uppercase list kind # defaulted to be kindList
    categories: list of category membership like "all" # optional
  validation: # optional
    openAPIV3Schema: OpenAPI schema # optional
  subresources: # optional
    status: {} # to enable the status subresource (optional)
    scale: # optional
      specReplicasPath: JSON path for the replica number in the spec of the
                        custom resource
      statusReplicasPath: JSON path for the replica number in the status of
                          the custom resource
      labelSelectorPath: JSON path of the Scale.Status.Selector field in the
                         scale resource
  versions: # defaulted to the Spec.Version field
  - name: version name
    served: boolean whether the version is served by the API server # defaults to false
    storage: boolean whether this version is the version used to store object
  - ...

定义之后提交, API server就开始可以进行验证了, API server验证这些字段是采用OpenAPI v3 schema 模式完成的, OpenAPI schema是基于JSON Schema standard , 使用JSON/YAML来表达一个schema , 下面是个例子

type: object
properties:
  apiVersion:
    type: string
  kind:
    type: string
  metadata:
    type: object
  spec:
    type: object
    properties:
      schedule:
        type: string
        pattern: "^\d{4}-([0]\d|1[0-2])-([0-2]\d|3[01])..."
      command:
        type: string
    required:
    - schedule
    - command
  status:
    type: object
    properties:
      phase:
        type: string
required:
- metadata
- apiVersion
- kind
- spec

上述schema其实是个JSON object , 每个JSON object中的spec 必须要求有2个字段:schedule and command , schedule匹配一个ISO date(这里是用正则表达式匹配)

Short Names and Categories

和本地资源一样, 自定义资源也可以用自己的短名称 , 比如namespace的简称就是ns , 我们可以用kubectl get ns来获取

查看所有可用的短名称 , 我们可以使用以下命令

$ kubectl api-resources
NAME                   SHORTNAMES  APIGROUP NAMESPACED  KIND
bindings                                    true        Binding
componentstatuses      cs                   false       ComponentStatus
configmaps             cm                   true        ConfigMap
endpoints              ep                   true        Endpoints
events                 ev                   true        Event
limitranges            limits               true        LimitRange
namespaces             ns                   false       Namespace
nodes                  no                   false       Node
persistentvolumeclaims pvc                  true       PersistentVolumeClaim
persistentvolumes      pv                   false       PersistentVolume
pods                   po                   true        Pod
statefulsets           sts         apps     true        StatefulSet
...

如何自定义短名称呢

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ats.cnat.programming-kubernetes.info
spec:
  ...
  shortNames:
  - at

当然还可以把他们加入一个categories

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ats.cnat.programming-kubernetes.info
spec:
  ...
  categories:
  - all

Printer Columns

Kubectl CLI 工具使用服务器端打印来呈现 kubectl get 的输出。这意味着它会向 API 服务器查询要显示的列和每一行中的值

通过 additionalPrinterColumns，自定义资源也支持服务器端打印列,它们被称为“ additional”

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ats.cnat.programming-kubernetes.info
spec:
  additionalPrinterColumns: (optional)
  - name: kubectl column name
    type: OpenAPI type for the column
    format: OpenAPI format for the column (optional)
    description: human-readable description of the column (optional)
    priority: integer, always zero supported by kubectl
    JSONPath: JSON path inside the CR for the displayed value

比如我们在自定义资源中添加如下字段

additionalPrinterColumns: #(optional)
- name: schedule
  type: string
  JSONPath: .spec.schedule
- name: command
  type: string
  JSONPath: .spec.command
- name: phase
  type: string
  JSONPath: .status.phase

之后我们使用kubectl来获取

1
2
3

$ kubectl get ats
NAME  SCHEDULER             COMMAND             PHASE
foo   2019-07-03T02:00:00Z  echo "hello world"  Pending

子资源状态

status状态子资源启用

例子:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
spec:
  ...
  versions:
  - name: v1alpha1
    served: true
    storage: true
  - name: v1beta1
    served: true
    subresources:
      status: {}

其中status中的字段要为空, 这是因为这些字段的填写是系统填写的

scale子资源

它允许我们查看和修改副本值

比如我们想用kubectl scale来扩展副本数量

1
2
3

$ kubectl scale --replicas=3 your-custom-resource -v=7
I0429 21:17:53.138353   66743 round_trippers.go:383] PUT
https://host/apis/group/v1/your-custom-resource/scale

在进行自定义资源定义的时候 , 我们可以

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
spec:
  subresources:
    scale:
      specReplicasPath: .spec.replicas
      statusReplicasPath: .status.replicas
      labelSelectorPath: .status.labelSelector
  ...

这样replica的值就由从GET返回的值填充到spec.replicas中去了

使用golang访问Custom resources

我们可以使用多种client来访问Custom resources

使用dynamic client
使用typed client

目前有2种方式生成类型化客户端
- 一种是由框架生成, 比如typed client和Kubebuilder提供的kubernetes-sigs/controller-runtime来生成
- 第二种方法是由k8s.io/client-go/kubernetes提供的client-gen生成

Dynamic Client

Dynamic Client不使用RESTMapper , 得手动添加类型

schema.GroupVersionResource{
  Group: "apps",
  Version: "v1",
  Resource: "deployments",
}

如果 REST 客户端配置可用，动态客户端可以在一行中创建

1	client, err := NewForConfig(cfg)

对给定的 GVR 的 REST 访问同样简单

1 2	client.Resource(gvr). Namespace(namespace).Get("foo", metav1.GetOptions{})

Typed Clients

常见的一个做法是把它放进一个约定的包里

1	pkg/apis/group/version

然后在types.go里面定义类型

每一个与 GVK 相对应的 Golang 类型都嵌入了来自包装 k8s.io/apimachinery/pkg/apis/meta/v1的 TypeMeta 结构。只包含 Kind 和 ApiVersion 字段

type TypeMeta struct {
    // +optional
    APIVersion string `json:"apiVersion,omitempty" yaml:"apiVersion,omitempty"`
    // +optional
    Kind string `json:"kind,omitempty" yaml:"kind,omitempty"`
}

此外还需要一个名称以及名称空间等等字段, 他们存储在一个名为 ObjectMeta 的结构体中，该结构体位在k8s.io/apimachinery/pkg/apis/meta/v1文件包中

type ObjectMeta struct {
    Name string `json:"name,omitempty"`
    Namespace string `json:"namespace,omitempty"`
    UID types.UID `json:"uid,omitempty"`
    ResourceVersion string `json:"resourceVersion,omitempty"`
    CreationTimestamp Time `json:"creationTimestamp,omitempty"`
    DeletionTimestamp *Time `json:"deletionTimestamp,omitempty"`
    Labels map[string]string `json:"labels,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
    ...
}

Kubernetes 顶级类型(即那些具有嵌入式 TypeMeta 和嵌入式 ObjectMeta 的类型，并且在本例中被持久化到 etcd 中)看起来非常相似，因为它们通常具有 spec 和status。看看这个来自 k8s.io/kubernetes/apps/v1/types.go 的部署示例

type Deployment struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec DeploymentSpec `json:"spec,omitempty"`
    Status DeploymentStatus `json:"status,omitempty"`
}

golang包的组织

Golang 类型通常放在一个名为 types.go 的文件中,放在包pkg/apis/group/version 中 , 除了这个文件还有其他文件

doc.go文件描述了 API 的用途，并包含了一些 package-global 代码生成标签

// Package v1alpha1 contains the cnat v1alpha1 API group
//
// +k8s:deepcopy-gen=package
// +groupName=cnat.programming-kubernetes.info
package v1alpha1

package version

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/runtime/schema"

    group "repo/pkg/apis/group"
)

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{
    Group: group.GroupName,
    Version: "version",
}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
    return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group
// qualified GroupResource
func Resource(resource string) schema.GroupResource {
    return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
    SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
    AddToScheme   = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
    scheme.AddKnownTypes(SchemeGroupVersion,
        &SomeKind{},
        &SomeKindList{},
    )
    metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
    return nil
}

之后zz_generated.deepcopy.go文件自动生成 , 它定义了custom resource Golang top-level types的deep-copy method , 关于如何用go的注释标签自动生成代码, 我们下一讲再说

通过client-gen生成typed client

在pkg/apis/group/version这个地方 , client generator client-gen在pkg/generated/clientset/versioned (老版本是/client/clientset/versioned里面生成)生成了一个typed client

它长的是这样

// Code generated by client-gen. DO NOT EDIT.

package versioned

import (
    discovery "k8s.io/client-go/discovery"
    rest "k8s.io/client-go/rest"
    flowcontrol "k8s.io/client-go/util/flowcontrol"

    cnatv1alpha1 ".../cnat/cnat-client-go/pkg/generated/clientset/versioned/
)

type Interface interface {
    Discovery() discovery.DiscoveryInterface
    CnatV1alpha1() cnatv1alpha1.CnatV1alpha1Interface
}

// Clientset contains the clients for groups. Each group has exactly one
// version included in a Clientset.
type Clientset struct {
    *discovery.DiscoveryClient
    cnatV1alpha1 *cnatv1alpha1.CnatV1alpha1Client
}

// CnatV1alpha1 retrieves the CnatV1alpha1Client
func (c *Clientset) CnatV1alpha1() cnatv1alpha1.CnatV1alpha1Interface {
    return c.cnatV1alpha1
}

// Discovery retrieves the DiscoveryClient
func (c *Clientset) Discovery() discovery.DiscoveryInterface {
   ...
}

// NewForConfig creates a new Clientset for the given config.
func NewForConfig(c *rest.Config) (*Clientset, error) {
    ...
}

这个clinet set由interface Interface 表示,并为每个版本提供对 API 组客户端接口的访问接口ー例如，下面的示例代码中的 cnatv1alpha1接口:

type CnatV1alpha1Interface interface {
    RESTClient() rest.Interface
    AtsGetter
}

// AtsGetter has a method to return a AtInterface.
// A group's client should implement this interface.
type AtsGetter interface {
    Ats(namespace string) AtInterface
}

// AtInterface has methods to work with At resources.
type AtInterface interface {
    Create(*v1alpha1.At) (*v1alpha1.At, error)
    Update(*v1alpha1.At) (*v1alpha1.At, error)
    UpdateStatus(*v1alpha1.At) (*v1alpha1.At, error)
    Delete(name string, options *v1.DeleteOptions) error
    DeleteCollection(options *v1.DeleteOptions, listOptions v1.ListOptions) error
    Get(name string, options v1.GetOptions) (*v1alpha1.At, error)
    List(opts v1.ListOptions) (*v1alpha1.AtList, error)
    Watch(opts v1.ListOptions) (watch.Interface, error)
    Patch(name string, pt types.PatchType, data []byte, subresources ...string)
        (result *v1alpha1.At, err error)
    AtExpansion
}

可以使用 NewForConfig 函数创建client set的实例

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/tools/clientcmd"

    client "github.com/.../cnat/cnat-client-go/pkg/generated/clientset/versioned"
)

kubeconfig = flag.String("kubeconfig", "~/.kube/config", "kubeconfig file")
flag.Parse()
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
clientset, err := client.NewForConfig(config)

ats := clientset.CnatV1alpha1Interface().Ats("default")
book, err := ats.Get("kubernetes-programming", metav1.GetOptions{})

Operator SDK and Kubebuilder的controller-runtime Client

下面是个例子

import (
    "flag"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes/scheme"
    "k8s.io/client-go/tools/clientcmd"

    runtimeclient "sigs.k8s.io/controller-runtime/pkg/client"
)

kubeconfig = flag.String("kubeconfig", "~/.kube/config", "kubeconfig file path")
flag.Parse()
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)

cl, _ := runtimeclient.New(config, client.Options{
    Scheme: scheme.Scheme,
})
podList := &corev1.PodList{}
err := cl.List(context.TODO(), client.InNamespace("default"), podList)

在上述中 client object的List()方法接受任何一个给定scheme中的runtime.Object , 将corev1.PodList这个go类型映射到GVK , 之后由GVR获取pod , 即通过 api/v1/namespace/default/pods*来获取schema.GroupVersionResource{“”, “v1”, “pods”}

原文作者：HAOJX

原文链接：haojianxun.github.io/2019/09/08/kuberntes开发教程系列(3)了解和编写Custom Resources/

发表日期：September 8th 2019, 12:00:00 am

更新日期：September 22nd 2019, 6:32:48 pm

Next Post

kubernetes开发教程(4)--自动代码生成
Previous Post

kuberntes开发教程系列(2):了解client-go是什么

CATALOG

1. Discovery Information
2. 自定义资源的编写
1. 2.1. Short Names and Categories
2. 2.2. Printer Columns
3. 子资源状态
1. 3.1. status状态子资源启用
2. 3.2. scale子资源
4. 使用golang访问Custom resources
1. 4.1. Dynamic Client
2. 4.2. Typed Clients
5. golang包的组织
1. 5.1. 通过client-gen生成typed client
2. 5.2. Operator SDK and Kubebuilder的controller-runtime Client

